Is there an argument for why AddToFileServer() should be removed? It's a breaking change and also the ability to add files to the file server at different points in the build scripts is a useful feature.

It is no longer needed, because an application can now simply build a CMake list with the files. It can manipulate the list any time. Details are fully unter application control, until eventually the final list is passed to DefineCAmkESVMFileServer(), which is called in the end when DeclareCAmkESRootserver()is called.
There seems no much benefit if this appending to a list is wrapped in a function, and I also doubt this is a real convenience improvement. Also, from the changes you can see things get much simpler internally then. We do not have to maintain state variables over calls. I thing that makes it worth changing this. But as you said, the downside is a breaking change. So we may want to have deprecation period to give people time to change things. But the change is quite trivial, as you can see from the VM examples.

It is no longer needed, because an application can now simply build a CMake list with the files.

The current way doesn't require an application to manage the list itself, it simply has to call the function each time it wants to add files. This is a common pattern that's all throughout CMake and is also followed by the camkes-tool CMake modules as well.

As far as I know there's no strong reason why the existing implementation is insufficient. Maybe there are some complicated apps that want to do more complicated build time processing and mutate the list of files in the file server after they've been added but these apps would be free to manage their own list before declaring the fileserver at the end also.

I can see you point and agree that it is convenient for applications. I can also see that having the API AddToFileServer() can have some advantages, and keeping it causes no harm.
However, this creates some internal complexity and corner cases, which I think could be avoided in the long run. It's much better easier dealt with on the application level. I also consider change of using the function to managing a list to be trivial CMake usage, so it seems not really worth having another layer here. Unless, there more that AddToFileServer() could do, e.g. more file type specific transformation convenience functions But even these might be better kept agnostic of this specific file server then and just be intermediate CMake targets.

Turns out we can also remove the convenience wrapper DeclareCAmkESVMRootServer() then, as this is no longer used. My long term suggestion would be deprecating and removing this also to get rid of convenience layers that seem not really helpful because they hide too much.